Web Application rewalls (WAFs) are used for protecting Web Applications from attacks such as SQL injection, cross-site request forgery, and cross-site scripting. As a result of the growing complexity of Web attacks, WAFs need to be tested and updated on a regular basis. There are various tools and techniques to verify the correct performance of a WAF. But most of the techniques are manual or use brute-force attacks, so su er from poor e cacy. In this work, we propose a solution based on Reinforcement Learning (RL) to discover malicious payloads, which are able to bypass WAFs. We provide an RL framework with an environment compatible with OpenAI gym toolset standards. The environment is employed for training agents to implement WAF circumvention tasks. The agent mutates the syntax of a malicious payload using a set of modi cation operators as actions, without changes to its semantic. Then, upon WAF's reaction to the payload, the environment ascertains a reward for the agent. Eventually, based on these rewards, the agent learns a suitable sequence of mutations for any malicious payload. The payloads, which bypass the WAF determine rules defects, which can be further used in rule tuning for rule-based WAFs. Also, it can enrich the machine learning-based WAFs datasets for retraining. We use Q-Learning, Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO) algorithms with the deep neural network. Our solution is successful in evading signature-based and machine learning-based WAFs. While our focus in this work is on SQL injection, the method can be simply extended to use for any string-based injection attacks.